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Abstract 

Version  1.0  of  the  Guide  to  the  Systems  Engineering  Body  of  Knowledge  (SEBoK)  was  released  in  September  2012.  Seventy 
authors  from  around  the  world  working  for  three  years  created  the  SEBoK.  This  paper  discusses  twelve  factors  that  were  critical 
to  the  successful  development  and  on-time  release  of  the  SEBoK.  The  approach  taken  is  elaborated,  and  lessons  learned  along  the 
way  are  highlighted.  These  experiences  and  recommendations  based  on  them  would  be  useful  to  any  collaborative  research  effort 
that  shares  one  or  more  of  the  characteristics  of  SEBoK  development. 
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1.  Introduction 

The  objective  of  this  paper  is  to  share  the  experiences  gained  during  the  development  and  successful  on-schedule 
release  of  version  1.0  of  the  Guide  to  Systems  Engineering  Body  of  Knowledge  (SEBoK).  The  paper  also  offers 
recommendations  based  on  those  experiences,  that  could  be  valuable  to  any  collaborative  research  effort  that  shares 
many  of  the  central  characteristics,  such  as  being  largely  a  volunteer  effort,  including  professional  society 
participation,  and  involving  a  large  number  of  authors  from  diverse  locations,  employers,  and  backgrounds. 
However,  the  objective  of  this  paper  is  not  to  elaborate  on  the  principles  and  methods  that  were  adopted  during  the 
development  of  SEBoK. 

In  September  2009,  the  Body  of  Knowledge  and  Curriculum  to  Advance  Systems  Engineering  (BKCASE™) 
project  was  established  with  primary  sponsorship  by  the  U.S.  Department  of  Defense  (DoD)  through  the  Systems 
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Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 
VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  OMB  control  number. 
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Engineering  Research  Center  (SERC).  It  was  a  university  led  partnership  between  Stevens  Institute  of  Technology, 
Hoboken,  NJ  and  Naval  Postgraduate  School,  Monterey,  CA,  with  support  from  various  other  universities, 
organizations,  government  agencies,  and  professional  societies.  The  objective  of  the  BKCASE  project  was  to  create 
and  deliver  two  products:  (1)  Guide  to  the  Systems  Engineering  Body  of  Knowledge  (SEBoK)  by  September  2012 
and  (2)  Graduate  Reference  Curriculum  for  Systems  Engineering  (GRCSE)  by  December  2012.  Both  these  products 
are  intended  for  worldwide  use,  available  for  public  use  at  no  cost.  At  the  time  of  writing  this  paper,  version  1.0  of 
SEBoK  had  been  released  on  September  14,  2012  [1]  and  version  1.0  of  GRCSE  was  on  track  for  release  on 
December  15,  2012. 

The  project  charter  of  the  BKCASE  project  [2]  summarizes  the  current  state  of  the  Systems  Engineering  (SE) 
discipline;  the  intent  behind  creating  the  SEBoK;  and  the  value  proposition  of  the  SEBoK: 

•  There  is  no  authoritative  source  that  defines  and  organizes  the  knowledge  of  the  SE  discipline. 

•  The  resulting  knowledge  gap  creates  unnecessary  inconsistency  and  confusion  in  understanding  SE. 

•  The  process  of  creating  the  SEBoK  will  help  build  community  consensus  on  the  boundaries  and  context  of  SE. 

•  Having  a  common  way  to  refer  to  SE  knowledge  will  facilitate  communication  among  systems  engineers  and 
provide  a  baseline  for  competency  models,  certification  programs,  educational  programs,  and  other  workforce 
development  initiatives  around  the  world. 

While  the  SEBoK  development  effort  is  significant,  foundational,  and  unique  for  the  discipline  of  SE,  it  may  be 
generally  characterized  as  below: 

•  International  author  team  of  70  members  from  10  different  countries; 

•  Collaborative  concurrent  authorship; 

•  Published  as  a  wiki  over  the  Internet  for  free  public  use; 

•  Confluence  of  academia,  industry,  government  agencies,  and  professional  societies; 

•  Convergence  of  different  backgrounds,  experiences,  philosophies,  approaches,  and  perspectives  on  SE; 

•  Multi-year  effort  (3  years  for  development  of  SEBoK  v.  1.0  to  be  followed  by  periodic  updates  and  releases); 

•  Large  scale  (primary  content  is  about  850  pages,  not  including  glossary  terms,  annotations,  acronyms,  etc.);  and 

•  Distributed  direct  funding  and  in-direct  funding. 

The  SEBoK  is  a  significant  contribution  to  the  discipline  of  systems  engineering,  and  is  a  valuable  resource  for 
systems  engineering  research.  In  addition  to  providing  informative  insight  to  its  many  users,  the  experiences  gained 
from  developing  the  SEBoK,  lessons  learned,  and  recommendations  reported  in  this  paper  would  also  be  valuable  to 
any  similar  development  effort  that  shares  one  or  more  of  the  above  characteristics. 

2.  Twelve  Critical  Factors 

Managing  the  development  of  the  SEBoK  since  initiation  in  2009  until  the  release  of  version  1.0  in  September 
2012  has  been  a  rich,  valuable,  and  rewarding  experience  to  everyone  involved,  particularly  to  the  authors  of  this 
paper:  the  core  editorial  team  of  the  SEBoK.  These  experiences  have  been  gathered  under  twelve  critical  factors. 
Listed  in  no  particular  order,  the  manner  in  which  each  of  the  following  factors  was  addressed  played  a  significant 
role  in  the  successful  on-schedule  release  of  SEBoK  v.  1.0:  (1)  Content  Sourcing  and  Organization;  (2)  Roles  and 
Responsibilities;  (3)  Collaboration:  Modes  &  Tools;  (4)  Content:  Style,  Quality,  and  Integration;  (5)  Internal  Review 
and  Editing;  (6)  Wiki  Environment;  (7)  Intellectual  Property  and  Copyright;  (8)  Outreach  and  Community  Review; 
(9)  Rollout  Activities;  (10)  Sponsorship;  (11)  Stewardship;  and  (12)  Updates  and  Future  Versions. 

In  the  following  sections,  the  approach  taken  for  each  of  the  above  factors  is  discussed  along  with  challenges 
faced  and  experiences  gained.  Lessons  learned  are  summarized  as  recommendations.  These  factors  are  neither 
exclusive  nor  do  they  indicate  any  sequence  of  steps  or  process.  There  is  significant  overlap  between  them,  and  they 
also  strongly  influence  each  other.  Further,  this  list  is  not  exhaustive,  but  does  represent  the  most  important  ones  in 
the  opinion  of  the  authors  of  this  paper.  This  offers  a  good  starting  checklist  for  any  comparable  effort. 
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2.1.  Content  Sourcing  and  Organization 

The  SEBoK  is  a  guide  to  the  SE  knowledge,  pointing  to  important  books,  papers,  and  websites,  and  not  an 
exhaustive  resource  that  captures  all  available  SE  knowledge  directly.  Therefore,  while  the  identification  of 
appropriate  SE  literature  was  critically  important,  the  balance  between  what  content  should  actually  be  included  in 
the  SEBoK  and  what  content  should  only  be  pointed  to  from  within  the  SEBoK  to  other  external  sources,  was  not 
easy  to  achieve. 

SE  Literature:  The  most  comprehensive  source  of  SE  knowledge  today  is  the  Systems  Engineering  Handbook  [3] 
prepared  by  the  International  Council  on  Systems  Engineering  (INCOSE).  In  fact,  the  initial  version  of  the  SE 
handbook  was  used  in  INCOSE’s  efforts  to  create  a  guide  to  the  systems  engineering  body  of  knowledge 
(G2SEBOK)  in  2002  [4]  that  was  subsequently  discontinued.  SE  literature  also  includes  many  standards  such  as 
ISO/IEC/IEEE  15288  [5],  models  such  as  CMMI  [6],  guidebooks  such  as  MITRE  SE  Guide  [7],  and  textbooks  such 
as  Systems  Engineering  and  Analysis  [8].  SE  related  articles  are  published  in  the  Systems  Engineering  Journal  and 
many  other  technical  journals. 

Organization  of  SEBoK  Content:  Two  other  bodies  of  knowledge  that  may  be  comparable  to  the  SEBoK  in 
content,  purpose,  and  usability,  are  the  Software  Engineering  Body  of  Knowledge  (SWEBOK)  [9]  and  A  Guide  to 
the  Project  Management  Body  of  Knowledge  (PMBOK®  Guide)  [10].  The  SEBoK  structure  was  initially  patterned 
on  the  INCOSE  SE  Handbook  and  the  ISO/IEC/IEEE  15288  standard,  but  the  SEBoK  structure  evolved 
significantly  over  time.  At  the  time  of  SEBoK  v.  1.0  release,  the  content  was  organized  in  a  manner  somewhat 
analogous  to  that  of  the  SWEBOK.  The  lowest  organizational  unit  in  the  SEBoK  is  a  “topic”.  Related  topics  are 
gathered  under  “knowledge  areas”,  and  related  knowledge  areas  are  gathered  under  seven  “parts”  of  the  SEBoK. 
There  are  26  knowledge  areas  and  112  topics  in  SEBoK  v.  1.0.  Every  part,  knowledge  area,  and  topic  of  the  SEBoK 
has  an  “article”  written  on  it. 

Organization  of  SE  Literature  within  SEBoK:  Due  to  the  nature  of  the  SEBoK,  it  was  imperative  that  every 
article  cite  and  point  to  relevant  SE  literature  that  would  provide  further  elaboration  and  detail.  However,  each 
SEBoK  article  also  included  sufficient  discussion  of  the  topic  to  provide  the  reader  a  reasonable  understanding  of 
the  topic  even  within  the  SEBoK.  The  references  for  each  article  are  organized  under  three  categories:  (1)  Works 
cited  include  all  references  directly  cited  in  the  SEBoK  article;  (2)  Primary  references  include  select  references 
that  are  key  to  the  topic  of  the  article,  and  are  strongly  recommended  for  further  reading  -  SEBoK  v.  1.0  includes 
224  primary  references;  and  (3)  Additional  references  include  other  related  literature  that  are  good  sources  of 
additional  information  beyond  the  primary  references  and  the  particular  sections  cited  in  that  SEBoK  article. 

Recommendations : 

•  Clearly  identify  the  critical  sources  for  content.  While  they  are  imperative  for  any  “body  of  knowledge”,  they 
are  equally  important  for  any  reference  resource. 

•  Establish  an  initial  structure  into  which  content  can  be  populated,  but  expect  the  structure  to  evolve.  Look  for 
similar  works  for  a  good  starting  point,  but  expect  and  allow  the  final  structure  to  emerge.  Using  an  analog 
from  agile  software,  expect  to  refactor  the  document  architecture  repeatedly  as  the  effort  progresses. 

2.2.  Roles  and  Responsibilities 

The  management  and  authoring  responsibilities  of  the  SEBoK  were  successfully  executed  by  a  diverse  set  of 
individuals.  They  contributed  individually,  as  part  of  a  small  group  or  groups,  as  leaders,  and  as  administrators. 
These  significant  roles  and  the  responsibilities  are  highlighted  here. 

Authors:  Authors  played  the  critical  role  of  identifying  the  right  mix  of  SE  literature  for  inclusion  in  the  SEBoK 
and  also  wrote  the  SEBoK  articles.  They  were  an  intentional  blend  of  academicians  and  industry  professionals  from 
the  Americas,  Europe,  Asia  and  Australia.  This  brought  in  a  variety  of  SE  backgrounds  and  perspectives  and  had 
two  other  significant  impacts:  (1)  the  SEBoK  could  be  created  keeping  a  global  audience  in  mind  and  (2)  the  time 
zones  and  multiple  native  languages  added  further  challenges  to  the  collaborative  effort.  Authors  were  continually 
recruited  to  the  SEBoK  author  team  based  on  discovered  gaps  in  requisite  expertise,  usually  relying  on  existing 
SEBoK  authors  to  find  new  recruits.  Authors  volunteered  their  time  for  the  SEBoK  effort  without  any  direct 
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monetary  compensation  from  the  BKCASE  project. 

Part  Team  Leads:  Authors  were  grouped  under  each  of  the  seven  parts  of  the  SEBoK,  with  one  of  them  serving 
as  the  lead  of  the  team.  This  ensured  better  communication,  collaboration,  and  effective  working  of  the  authors.  The 
part  team  lead  (PTL)  was  responsible  for  coordinating  the  work  of  the  authors  and  for  meeting  the  project  deadlines. 

Core  Team:  The  core  team  consisted  of  six  members  from  Stevens  Institute  of  Technology,  Naval  Postgraduate 
School,  and  Sevatec,  a  private  company.  It  provided  overall  leadership  and  played  three  significant  roles:  (1)  a  high- 
level  management  team  that  provided  overall  guidance,  support,  event  planning,  and  execution  of  every  activity  and 
event  of  SEBoK  development,  (2)  an  editorial  team  that  was  responsible  for  ensuring  the  quality  of  the  content  of 
the  SEBoK,  including  the  glossary,  acronyms,  and  primary  references,  and  (3)  a  publication  team  that  handled 
packaging  and  release  of  the  SEBoK.  The  core  team  planned  project  milestones,  supported  every  part  team  lead, 
organized  workshop  and  meetings  of  the  authors,  spearheaded  various  outreach  and  publicity  activities,  provided 
most  of  the  administrative  and  technical  support  to  the  entire  project,  and  managed  all  the  legal  and  related  issues  of 
the  SEBoK.  The  core  team  also  took  a  lead  in  forming  specialty  teams  at  various  stages  as  unique  needs  arose.  The 
core  team  (and  the  BKCASE  project)  was  led  by  a  principal  investigator  and  assisted  by  a  co-principal  investigator, 
who  had  final  authority  and  responsibility  for  the  project. 

SERC  Office:  The  BKCASE  project  was  one  of  the  research  tasks  of  the  Systems  Engineering  Research  Center 
(SERC),  which  is  a  DoD  University  Affiliated  Research  Center  managed  by  Stevens  Institute  of  Technology.  The 
SERC  provided  back-end  support,  funded  most  of  the  core  team’s  time  and  some  author  travel,  and  acted  as  a 
liaison  for  the  primary  sponsor. 

Recommendations: 

•  Authors  and  their  dedication  play  an  enormous  role  in  project  success.  Find  technically  qualified  authors  with 
the  right  collaborative  spirit,  a  willingness  to  work  as  a  team  and  honor  their  roles,  adhere  to  deadlines,  deal 
with  the  difficulties  of  a  large  diverse  team,  and  maintain  a  sense  of  humor  throughout  a  long  effort.  This  will 
not  be  easy,  but  is  critical.  How  well  author  candidates  will  fit  into  the  team,  must  also  be  considered. 

•  A  core  team  is  extremely  important  to  manage  and  execute  a  large-scale  project.  The  size  and  composition  of 
the  core  team  would  depend  on  a  number  of  factors,  but  a  lean  team  is  recommended. 

2.3.  Collaboration:  Modes  &  Tools 

A  number  of  modes  and  tools  were  employed  to  support  collaboration  and  communication. 

Emails,  Mailing  Lists,  and  Cloud  Storage:  Thousands  of  emails  were  exchanged  between  the  authors,  core  team 
members,  and  others  involved  with  the  SEBoK.  A  mailing  list  of  the  entire  group  of  SEBoK  authors  was  used  for 
bulk  communication.  Cloud  space  was  utilized  for  online  storage  and  sharing  of  files  between  the  authors,  within 
part  teams,  and  within  the  core  team. 

Virtual  Meetings:  Because  the  SEBoK  authors  were  geographically  dispersed,  virtual  meetings  using  Internet- 
based  technologies  were  both  common  and  essential.  WebEx  is  a  web  conferencing  and  collaboration  product  used 
by  BKCASE.  WebEx  sessions  were  the  most  common  method  of  meeting  for  the  core  team  (that  met  every  week  for 
the  entire  length  of  the  project),  and  the  part  teams. 

Face-to-face  Meetings:  Electronic  communication,  while  essential,  did  not  substitute  for  periodic  face-to-face 
meetings  that  were  essential  to  take  stock  of  progress,  to  build  team  rapport,  and  for  deliberations  and  decision¬ 
making  on  important  issues.  Workshops  conducted  every  three  or  four  months,  brought  together  20  to  30  authors  for 
two  or  three  days.  The  first  workshop  was  held  at  Naval  Postgraduate  School  in  December  2009  and  the  twelfth 
(last)  workshop  was  held  at  Stevens  Institute  of  Technology  in  October  2012.  Other  workshops  were  held 
throughout  the  US  and  Europe,  often  co-located  with  other  international  SE  events  and  conferences.  At  these 
workshops,  the  authors  made  significant  decisions  that  shaped  the  structure  and  content  of  the  SEBoK.  The  value  of 
the  workshops  was  increased  by  well-planned  agendas,  and  each  meeting  had  an  announced  set  of  objectives. 
Thorough  and  detailed  meeting  reports  assured  that  the  results  were  disseminated  to  parties  wsho  could  not  attend. 


Recommendations : 
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•  A  variety  of  online  collaboration  tools  (emails,  mailing  lists,  cloud  storage,  virtual  meetings)  is  required  for  any 
large  collaborative  effort.  It  is  important  that  all  participants  are  able  to  access  and  use  these  tools.  Access 
restrictions  imposed  by  some  organizations  can  pose  special  problems  that  must  be  addressed;  e.g.,  some 
companies  would  not  allow  WebEx  to  be  used  inside  their  corporate  firewalls. 

•  Hold  face-to-face  meetings.  To  minimize  cost  and  facilitate  easy  travel,  meeting  locations  should  be  based  on 
where  the  authors  live  and  where  events  of  common  interest  are  held. 

2.4.  Content  Style,  Quality,  and  Integration 

The  SEBoK  is  not  just  a  collection  of  individual  contributions  by  various  authors.  It  is  a  single  product  that  was 
collaboratively  developed  by  70  authors.  Ensuring  that  the  SEBoK  looks,  reads,  and  feels  like  a  single  body  of  work 
was  important. 

Article  Format:  To  ensure  consistency  and  quality  across  all  SEBoK  articles,  the  approximate  length  and 
expected  coverage  of  articles  were  specified.  Articles  had  to  be  descriptive,  not  prescriptive,  reflect  the  opinions  and 
views  of  several  authors,  and  had  to  objectively  point  to  relevant  SE  sources  for  further  details.  Moreover,  authors 
would  frequently  comment  on  the  work  of  others  and  volunteer  to  help  out  wherever  needed.  For  this  reason, 
individual  articles  are  not  attributed  to  individual  authors;  i.e.,  there  are  no  bylines  for  the  articles. 

Style  Guide:  The  core  team  prepared  a  comprehensive  style  guide  for  all  authors  to  use.  The  style  guide 
explained  the  expected  format  of  an  article,  how  that  article  should  link  to  others  within  the  SEBoK,  how  to  define 
and  reference  glossary  terms,  cite  references,  and  format  figures  and  tables.  The  style  guide  was  continually  updated 
as  the  project  progressed,  and  it  also  provided  a  basis  for  editing  of  the  SEBoK. 

Integration  of  Content:  While  the  style  guide  helped  maintain  a  consistent  feel  across  the  SEBoK,  the  content 
needed  to  be  integrated  at  a  deeper  level.  The  usage  and  perceived  understanding  of  a  term  should  be  consistent 
across  the  SEBoK,  irrespective  of  who  wrote  the  article  or  where  in  the  SEBoK  the  article  fit.  In  some  ways,  this 
was  the  most  challenging  aspect  of  the  writing,  given  the  many  different  perspectives  and  schools  of  thought  within 
the  SE  discipline.  Groups  were  formed  just  to  look  over  the  SEBoK  articles  and  look  for  places  where  terms  were 
used  inconsistently  with  each  other  and  with  the  glossary  definitions.  This  was  very  challenging  and  only 
imperfectly  implemented  in  version  1.0.  Subsequent  SEBoK  versions  will  continue  to  strengthen  content 
integration. 

Recommendations: 

•  In  any  effort  involving  a  large  number  of  contributors,  establish  detailed  guidance  on  content  format  to  ensure 
consistency. 

•  Establish  continuously  applied  mechanisms  to  ensure  consistency  not  just  in  the  style  but  also  in  the  semantics 
of  the  content  to  ensure  that  readers  gain  a  consistent,  unambiguous,  and  valuable  understanding. 

2.5.  Review  and  Editing 

In  spite  of  detailed  style  guidance  and  other  instructions,  a  thorough  review  and  editing  of  all  content  was 
essential.  Individual  authors  have  their  own  writing  styles,  and  some  of  them  may  focus  attention  on  the  content  per 
se  and  not  so  much  on  formatting  related  issues. 

Internal  Review  and  Editing:  Authors  and  the  respective  part  team  leads  did  their  own  review  and  editing  after  an 
article  was  written  to  ensure  compliance  with  the  style  guide  and  other  instructions.  However,  the  core  team 
performed  the  bulk  of  the  review  and  editing.  Activities  included  proofreading  for  typographical  errors;  citing  and 
listing  references  in  the  correct  format;  ensuring  proper  application  of  both  internal  and  external  links,  ensuring 
proper  formatting  of  tables  and  figures;  checking  consistent  usage  of  acronyms  and  glossary  terms;  ensuring  proper 
capitalization;  and  verifying  correctness  and  validity  of  the  content.  Because  so  much  of  getting  the  style  correct 
was  manual  and  often  very  detailed,  the  authors  simply  did  not  generally  do  this  well  even  when  they  tried. 

Technical  Editing:  Professional  technical  editors  were  hired  to  aid  the  editing  of  all  content.  Every  SEBoK  article 
underwent  at  least  one  round  of  technical  editing.  Articles  that  required  extra  rounds  of  technical  editing  were 
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flagged  during  internal  review  either  by  authors  or  by  the  core  team.  Most  often,  core  team  editing  and  technical 
editing  happened  in  parallel,  but  both  groups  working  on  the  same  article  at  any  point  in  time  was  avoided. 

Recommendations: 

•  More  important  that  developing  a  style  guide,  verify  that  the  guidance  is  being  followed. 

•  In  addition  to  all  the  internal  efforts  in  reviewing  and  editing,  engage  suitable  external  professional  help  for 
large  bodies  of  work. 

2.6.  Wiki  Environment 

One  of  the  most  useful  characteristics  of  the  SEBoK  is  its  wiki  environment.  While  SEBoK  v.  0.25,  was  released 
in  a  Portable  Document  Format  (PDF)  document,  subsequent  interim  versions  and  SEBoK  v.  1.0  were  released  as  a 
wiki.  This  dramatically  shifted  how  content  was  generated,  edited,  reviewed,  and  published  [11],  and  introduced 
significant  workload  into  the  project  to  establish  a  stable  wiki  environment  that  was  tailored  to  how  the  SEBoK 
authors  wanted  it  to  appear  and  have  specific  navigation  and  other  features. 

Content  Generation  and  Editing:  The  wiki  is  designed  for  collaborative  development,  providing  a  convenient 
platform  for  easy  and  quick  content  generation.  Wiki  editors  need  to  be  familiar  with  the  technical  language  that  is 
used  to  specify  elements  of  a  page  such  as  section  titles,  formatting,  tables,  etc.  Every  SEBoK  author  underwent 
training  on  the  wiki  environment  and  its  features.  Though  each  author  would  directly  work  on  only  a  few  of  the 
SEBoK  articles,  permissions  allowed  an  author  to  edit  any  wiki  page.  This  eliminated  administrative  overheads  of 
assigning  specific  page  permissions  to  specific  authors,  and  worked  reasonably  well  among  the  author  community, 
who  had  to  work  cooperatively  to  avoid  degrading  the  articles. 

Tools  and  Controls:  The  wiki  environment  provided  a  number  of  tools  to  help  content  generators,  editors,  and 
administrators.  For  example,  it  is  easy  to  find  all  wiki  pages  that  link  to  a  particular  page,  or  to  know  the  entire 
editing  history  of  a  particular  page.  Permissions  for  each  group  of  users  can  be  specified;  therefore  permission  to 
create  or  delete  pages  can  be  restricted  to  a  few  administrators  only,  while  all  authors  may  be  able  to  edit  content  in 
existing  pages.  It  is  possible  to  maintain  a  private  site  that  is  accessible  only  to  authorized  users  where  all 
development  and  testing  can  be  done  before  releasing  the  site  to  the  public. 

User  Experience:  Wiki  users  can  do  more  in  a  wiki  environment  than  in  a  traditional  website,  and  much  more 
than  in  a  PDF  document;  SEBoK  includes  several  navigation  features.  Links  from  within  articles  takes  a  user  to 
other  related  articles  or  glossary  pages.  Most  importantly,  a  user  can  quickly  provide  feedback  on  the  wiki  page. 

Recommendations : 

•  Wiki  is  a  powerful  versatile  environment  for  collaborative  development,  but  before  letting  contributors  to  begin 
working  on  it,  set  up  the  overall  structure,  features,  tools,  accounts,  and  permissions. 

•  Publishing  on  a  wiki  is  unlike  publishing  a  book  that  would  be  stable  until  the  next  edition.  Keep  the  content 
dynamic  and  make  frequent  updates  as  required. 

•  A  wiki  environment  is  fairly  complex,  requiring  special  skills  to  maintain  and  customize.  Put  someone  in 
charge  of  the  wiki  environment,  supported  by  people  who  understand  the  underlying  information  technology. 

2. 7.  Intellectual  Property  and  Copyright 

Intellectual  property  and  copyright  issues  needed  to  be  addressed  for  two  kinds  of  content:  (1)  content  sourced 
from  other  literature  for  use  within  the  SEBoK  and  (2)  distribution  and  use  of  content  generated  by  the  SEBoK 
authors. 

Content  Sourced  from  Elsewhere:  Being  a  guide  to  the  body  of  knowledge,  the  SEBoK  references  a  large 
number  of  SE  resources  including  books,  articles,  and  websites.  Citing  these  resources  in  a  proper  ethical  manner 
without  plagiarizing  the  content  but  ensuring  fair  use  is  mandatory.  In  addition,  SEBoK  articles  also  reference 
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figures,  illustrations,  and  tables  that  would  benefit  the  SEBoK  reader  if  they  were  included  directly  in  the  SEBoK. 
This  requires  using  copyrighted  images  and  other  content.  Each  publisher  of  textbooks  or  journal  papers  has  varying 
rules  for  protecting  their  intellectual  property  and  extending  copyright  permissions.  In  many  cases,  SEBoK  authors 
who  wished  to  include  content  that  they  themselves  had  authored  outside  the  BKCASE  project,  did  not  themselves 
own  the  copyright.  Rather,  other  publishers  owned  the  copyright.  The  procedure,  time  and  effort  involved  varied 
significantly  between  publishers.  A  full  list  of  all  copyrighted  images  being  used  in  the  SEBoK  was  maintained,  and 
copyright  permissions  were  obtained  for  all  of  them.  Further,  most  publishers  required  standard  phrases  to  be 
included  along  with  the  figure  or  table  in  the  SEBoK,  which  were  include  in  the  caption  of  the  images  when 
applicable.  Management  of  the  third  party  intellectual  property  was  on  the  critical  path  for  the  project,  and  a 
significant  risk  to  on-time  completion.  This  risk  was  increased  when  SEBoK  shifted  from  print  media  to  wiki 
media,  and  had  to  re-secure  permissions.  Institutional  lawyers  were  risk  -averse  and  not  familiar  with  the  nuances 
of  permissions  in  the  complicated  publishing  scheme,  and  their  review  took  much  longer  than  anticipated. 

Author-Generated  Content:  The  authors  wrote  the  SEBoK  articles  as  part  of  SEBoK  development  effort, 
including  a  number  of  original  figures  and  tables.  The  authors  signed  over  the  copyright  of  their  work  to  Stevens 
Institute  of  Technology,  which  holds  a  compilation  copyright  to  the  SEBoK.  The  SEBoK  is  published  as  a 
compilation,  including  the  third  party  material,  under  the  terms  of  a  Creative  Commons  License  (CC  BY-NC-ND 
3.0)  [12].  This  license  does  not  permit  commercial  use  of  the  SEBoK.  Adoption  of  this  standard  licensing  scheme 
greatly  simplified  the  legal  review. 


Recommendations: 

•  Identify  all  copyrighted  third  party  material  early  in  the  development.  Begin  discussions  early  with  the 
respective  publishers.  There  will  be  many  twists  and  turns  in  getting  all  the  needed  permissions.  Allow  plenty  of 
time  to  sort  out  issues  as  they  emerge. 

•  Specify  how  intellectual  property  of  individual  authors  will  be  handled  and  get  signed  agreement  from  each 
author  when  he  or  she  joins  the  project. 

2.8.  Outreach  and  Community  Review 

Community  engagement  and  review  was  essential  to  ensure  that  the  SEBoK  is  relevant,  useful  and  acceptable. 
For  this  reason,  as  well  as  to  help  pace  the  author  team,  the  project  released  three  interim  versions  of  the  SEBoK: 
version  0.25  was  a  prototype,  version  0.5  was  intended  for  early  adopters,  and  version  0.75  was  a  pre-final  version. 

Conferences:  Annual  systems  engineering  conferences,  including  the  INCOSE  International  Symposium, 
Conference  on  Systems  Engineering  Research  (CSER)  and  the  annual  systems  engineering  conference  of  National 
Defense  Industrial  Association  (NDIA),  were  key  events  where  the  SEBoK  was  featured.  The  SEBoK  authors, 
usually  lead  by  the  core  team,  presented  papers  and  delivered  presentations  in  each  of  these  conferences  from  the 
beginning  of  the  SEBoK  effort.  In  addition,  tutorials  were  conducted  since  SEBoK  version  0.5  was  released  and 
panels  were  held  periodically  on  different  aspects  of  the  SEBoK. 

Other  Outreach:  In  addition  to  conferences,  authors  presented  the  SEBoK  at  the  INCOSE  International 
Workshops,  wrote  articles  for  INCOSE  Insight,  and  delivered  informational  talks  at  various  other  meetings  and 
gatherings.  Together,  these  efforts  exposed  the  SE  community  to  the  SEBoK  well  before  the  release  of  version  1.0. 

Community  Review:  The  SEBoK  authors  sought  three  rounds  of  community  feedback.  While  outreach  activities 
spread  the  word  about  SEBoK,  the  interim  releases  of  SEBoK  provided  an  opportunity  for  the  community  to  provide 
direct  feedback  comments  and  suggestions  on  the  content.  A  list  of  reviewers  was  initially  created  based  on 
recommendations  from  the  author  team  for  a  closed  review  of  version  0.25.  Over  time,  this  list  expanded  to  over  200 
reviewers  for  public  review  of  versions  0.5  and  0.75.  Reviewers  provided  their  feedback  through  pre-designed 
forms,  emails  and  in  separate  documents.  In  addition,  with  the  transition  to  the  wiki  environment,  some  reviewers 
offered  feedback  directly  on  the  wiki  site.  Following  the  release  of  a  SEBoK  version,  review  comments  were 
accepted  until  a  specific  time.  After  that,  all  comments  were  compiled  and  the  author  team  reviewed  them.  The 
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resulting  actions  varied  from  accepting  a  review  comment  and  making  the  suggested  change  to  deferring  the 
suggested  action  to  future  versions  of  the  SEBoK  or  simply  rejecting  a  comment  for  some  good  and  valid  reason. 
Each  version  of  the  SEBoK  reported  the  adjudication  of  comments  received  on  the  previous  version. 

Recommendations: 

•  Spread  the  word  repeatedly  through  a  variety  of  channels,  beginning  long  before  a  useful  version  of  the  product 
is  available  and  continuing  through  final  release. 

•  Gather  review  comments  from  the  target  audience  during  development  and  adjudicate  each  comment  to  ensure 
that  all  comments  are  properly  considered,  and  demonstrate  to  the  reviewers  and  the  broader  community  that 
review  comments  are  incorporated  into  the  product. 

2.9.  Roll  Out  Activities 

Preparing  a  SEBoK  version  for  release  involved  many  activities,  such  as  ensuring  all  the  links  on  the  wiki  were 
valid,  preparing  press  releases,  and  preparing  emails  for  reviewers,  sponsors,  and  authors. 

SEBoK  Release:  Since  moving  to  the  wiki  platform,  there  were  two  SEBoK  sites  that  co-existed:  (1)  A  public  site 
that  was  visible  across  the  Internet,  and  (2)  a  private  protected  site  where  authors  were  developing  the  next  version 
of  the  SEBoK.  Releasing  the  SEBoK  involved  making  the  development  site  public  and  taking  down  what  had 
previously  been  the  public  site.  This  involved  correcting  all  the  addresses  and  links  to  webpages.  At  any  point,  the 
public  site  could  not  be  edited  (except  for  wiki  administrators),  but  would  be  open  to  review  comments  by  any 
visitor.  As  release  neared,  the  authors  were  not  allowed  to  make  changes  in  the  wiki.  The  core  team  would  a  spend  a 
few  days  together  discussing  release  related  issues  and  performing  final  editing  of  the  entire  SEBoK,  which  would 
then  be  ready  for  release. 

Announcements:  Immediately  following  each  release,  a  number  of  notifications  were  sent  out.  The  authors  were 
informed  of  this  release,  thanking  them  for  their  efforts  and  requesting  them  to  further  spread  the  word.  All  sponsors 
and  supporting  agencies  and  professional  bodies  were  also  informed  of  the  release.  Official  press  releases  were 
issued  and  featured  in  websites  and  other  publications.  Emails  were  also  sent  out  to  SEBoK  reviewers  and  other 
members  of  the  SE  community  announcing  the  SEBoK  release.  All  these  communications  and  the  respective 
recipients  were  prepared  and  identified  in  advance  of  the  actual  release  date  and  time. 

Recommendations: 

•  Spread  information  about  a  release  widely  and  concurrent  with  the  release.  Be  sure  to  thank  all  those  involved 
in  the  project,  including  authors,  reviewers,  and  sponsors. 

2.10.  Sponsorship 

Two  types  of  funding  have  supported  the  SEBoK  development  and  other  related  activities. 

Direct  Funding:  The  office  of  the  Deputy  Assistant  Secretary  of  Defense  for  Systems  Engineering  (DASD/SE), 
DoD  sponsor  for  the  BKCASE  Project,  has  provided  substantial  funding  for  the  SEBoK  through  the  SERC.  They 
supported  SEBoK  leadership  (core  team)  manpower,  technical  and  administrative  infrastructure,  and  travel  support 
to  the  core  team  and  some  authors  to  attend  workshops,  conferences,  and  other  events.  However,  DASD/SE  did  not 
influence  the  content  of  the  SEBoK,  composition  of  the  author  team,  or  the  development  and  release  mechanisms. 
DASD/SE  provided  this  support  because  they  believe  that  the  expected  benefits  to  the  broad  SE  community  will 
equally  benefit  DoD.  Further,  they  believe  that  taking  a  hands-off  approach  to  SEBoK  development  and  content  will 
yield  the  best  product. 

Indirect  Funding:  Virtually  every  organization  that  was  represented  in  the  SEBoK  author  team  indirectly  funded 
the  SEBoK  activity.  Authors  volunteered  their  time;  their  parent  organizations  permitted  use  of  their  time  and 
resources  for  SEBoK.  Further,  many  organizations  funded  author  travel  to  meetings,  workshop,  and  conferences. 
For  every  dollar  invested  by  DoD,  it  is  estimated  that  others  invested  approximately  two  dollars. 
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Recommendations : 

•  Identify  the  total  funding  required  early,  including  different  types  of  funding.  Line  up  the  investors  in  the  effort. 

•  Establish  constraints  and  expectations  for  investment  and  ensure  all  parties  understand  them.  Continuously 
work  to  meet  all  such  constraints  and  expectations. 

2.11.  Stewardship 

While  DASD/SE,  DoD,  was  the  primary  sponsor  for  the  development  effort,  beginning  January  2013,  SEBoK 
will  migrate  to  a  shared  governance  structure  between  INCOSE,  the  Institute  of  Electrical  and  Electronic  Engineers 
-  Computer  Society  (IEEE-CS),  and  the  SERC.  These  three  organizations  would  be  the  stewards  of  the  two 
products  of  the  BKCASE  project  -  SEBoK  and  GRCSE;  forming  a  governing  board  that  will  guide  and  oversee  the 
maintenance  of  the  current  versions  and  development  of  future  versions. 

Engagement  of  Stewards:  Representatives  of  INCOSE  and  IEEE-CS  engaged  the  BKCASE  leadership  from  the 
very  beginning  and  participated  in  SEBoK  development.  Both  organizations  contributed  authors  and  held  regular 
meetings  with  the  SEBoK  leadership  to  establish  expectations  and  to  shape  governance  post  version  1.0.  Draft 
agreements  were  circulated  months  before  version  1.0  was  released  to  facilitate  vetting  within  the  three  steward 
organizations.  The  structure  of  the  editorial  board  and  mechanisms  for  future  releases  were  discussed.  Current 
SEBoK  leadership  will  continue  and  DASD/SE  will  continue  to  fund  transition  activities,  but  at  a  lower  level  than 
was  available  for  development. 

Community  and  Author  Engagement:  Despite  the  anticipated  changes  in  the  governance,  the  SEBoK  will 
continue  to  rely  heavily  on  volunteer  authors  and  reviewers  from  across  the  SE  community.  Over  time,  new  authors 
will  join  the  team,  seasoned  authors  will  leave,  and  the  underlying  infrastructure  to  support  collaboration  will 
evolve.  Ensuring  continuous  engagement  of  the  community  and  relevant  authors,  and  generating  and  publishing 
subsequent  versions  of  the  SEBoK  will  remain  challenging. 

Recommendations : 

•  Maintain  a  life-cycle  perspective  of  the  product.  Resources  required  for  this  sustainment  (manpower,  funding, 
infrastructure)  must  be  identified  and  obtained  in  advance. 

•  Identify  and  resolve  early  among  the  stewards,  any  differences  in  expectations  and  future  responsibilities. 


2.12.  Updates  and  Future  Versions 

Interim  versions  prior  to  release  of  SEBoK  v.  1.0  played  a  significant  role  in  shaping  the  structure  and  content. 

Pre-SEBoK  v.  1.0  Releases:  The  three  interim  developmental  releases  of  SEBoK,  v.  0.25,  0.5,  and  0.75,  created 
breakpoints  to  introduce  changes  in  the  architecture  and  content.  Review  comments,  as  well  as  the  addition  of  new 
authors  and  deliberations  during  workshops  and  other  meetings,  drove  most  of  the  updates  to  the  SEBoK.  Major 
structural  changes  were  performed  in  SEBoK  v.  0.5,  but  nearing  the  release  of  SEBoK  v.  1.0,  structural  changes 
were  minimal.  Changing  the  structure  meant  modifying  the  Table  of  Contents,  updating  links  to  and  from  the 
affected  articles,  and  checking  for  inconsistent  content.  While  authors  were  given  permission  to  make  content 
changes  within  an  article,  only  wiki  administrators  could  make  structural  changes,  which  required  the  approval  of  a 
change  management  board.  Proposed  changes  to  the  architecture  faced  extra  scrutiny  because  of  their  impact. 

Post-SEBoK  v.  1.0  Releases:  SE  is  a  rapidly  changing  discipline,  and  it  is  imperative  to  keep  the  content  of  the 
SEBoK  current  and  valid,  perpetually.  Accordingly,  post  SEBoK  v.  1.0  releases  have  been  planned  for  the  future, 
and  the  next  immediate  version,  SEBoK  v.  1.0.1,  has  already  been  released  on  November  30,  2012  which  includes 
minor  changes  and  corrections  to  SEBoK  v.  1.0.  Similar  minor  updates  will  produce  SEBoK  v.  1.0.x  versions,  as 
needed.  Every  six  months,  more  significant  updates  will  produce  SEBoK  v.  1.x  versions.  SEBoK  v  2.0  is  unlikely  to 
be  produced  until  2015  or  2016. 
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Recommendations : 

•  Continually  seek  feedback  from  the  end  users  to  identify  deficiencies  and  opportunities  for  improvements. 

•  Establish  a  governance  structure  to  ensure  the  product  will  be  refreshed  commensurate  with  the  rate  of  change 
in  the  field  or  other  relevant  factors.  Plan  for  new  releases  as  required. 

3.  Conclusion 

Some  BKCASE  authors  have  participated  in  other  large,  collaborative  writing  projects  before,  but  the  scale  of  the 
SEBoK  and  the  magnitude  of  the  effort  broke  new  ground  for  the  entire  team.  The  factors  discussed  in  this  paper 
may  not  constitute  many  surprises  in  hindsight.  But  it  is  strongly  recommended  that  leaders  and  participants  of 
similar  projects  explicitly  identify  and  address  these  critical  factors  from  the  very  beginning. 
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